DOCUMENT; RESyME 



ED 227 0S7 

AUTHOR 
TITLE 

PUB DATE 
NOTE 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



SP 021 530 



IDENTIFIERS 



Denton r Jon J.; Ash, Michael J. 

Graduate Program Evaluation Employing a Status Survey 

and Matrix Scores. 

[78] 

18p. 

Reports - Research/Technical (143) 
MF01/PC01 Plus Postage. 

Curriculum Evaluation ; Curriculum Study Centers; 
Educational Psychology: *Education Majors; Education 
Work Relationship; Employment Patterns; *Graduates; 
*Graduate Study;' Graduate Surveys; Higher Education; 
*Program Ef f feet iveness ; Program Evaluation; Relevance 
(Education); Teacher Education ; *Teacher Education 
Programs . - 4 ' 

Texas A and M University 



from graduate programs in 
id in Educational Psychology 
•mine graduates '■ employment 



ABSTRACT 

A survey of former students 
Educational Curriculum and Instruction an< 
at Texas A & M University sought to determine gi 
patterns and professional profiles and their perceptions concerning 
the. effectiveness of their programs. Fifty-two percent of the 
graduates responded to mailed questionnaires, providing information 
on current professional role r membership in professional 
organizations, publications, presentations, and recognitions. They 
also rated the necessity and quality of service or instruction 
provided by the programs. Findings indicated that: (1) The majority 
of respondents were engaged in ^professional education; (2) Doctoral 
recipients were mbre active in professional associations; (3) 
Respondents, with doctorates were less critical of their programs than 
were their counterparts; and (4) Where program limitations were seen, 
graduates perceived the quality of the curriculum associated with 
certain services/skills to be limited given the relative importance 
of the skill, Program components found to be in need of revision" are 
listed, -and implications of the ^study are discussed. (JD) 
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A follow-up survey was conducted with graduates from four jgraduate 
programs in Educational Curriculum and Instruction and Educational Psychology. 
Status data f rom f the graduates revealed considerable professional activity and 
productivity of graduates. Additional data from the instrument w£re combined 
to reveal perceived progYam deficiencies. A'l 1 four programs examined were 
perceived to be deficient in three or more curricular components by these 
graduates . , 
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( Survey research may be directed toward identifying relationships 
among important variables. For example, an investigator may seek an answer 
to the question, "Is 'post -graduation vocational success perceived to be 
related to the graduate curricul urn?" \ In ofder'to answer this question, more 
sophisticated techniques and procedures must be implemented then merely 
measuring current success and attitude of recent graduates. Status surveys 
on the other hand tend to be descriptive, gathering facts -like median income 
of graduates, availability of resources,' etc. The purpose of such surveys 
lis generally to determine present status on variables of interest^ and not 
to stCidy inter-relations among these variables (Kerl inger7j^4) . * 

A review of numerous baccalaureate degree, fol low-up survey^ conducted 
by colleges and departments of education has shown that two types of surveys 
are typically conducted/ One type of survey asks broad questions related 
^o quality of advisement and counseling, personal valuing af 'specific courses, 
quality of faculty, quality of curriculum agd quality of instructional j • ■ 
methods (Univ. ofjNebraska, 1 976; EPSY-TAMU, 1 975; Devlin?- 1971). Ofher * 
ujrvey^ representing a status type follow-up have sought biographic information 
only, for example, teaching fields completed, teaching experience, annual 
earnings, honors bestowed, professional experiences, etc (fields, 1976; 
^enscfrl ing and Pope, 1 972; Miller, 1 974). ^ r ' 

While these surveys illustrate the status function adequately, the 
relational function of survey research is not as evident in these efforts. 
However, two-scaled items and open ended items used in other program* > 




evaluations (Rosser and Denton, 1977; College of Education Final Report, 
1977) appear to accommodate the relational function rather well. Instruments 
in these latter evaluation projects contained two Likert-type scales 
referenced to a single Item. Information was soug&f on the value of 



eric 
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different program componertts on one scale while a rating of quality of 
service provided for those components was sought oKthe second scale. 
These multiple responses enabled an empirical relationship to be determined 

for each program component. . . ^ 

» *" * 

v Purpose " , 

Given the aforementioned literature and available resources, a college 

level decision was made to conduct a follow-up survey of select graduate 

education programs at Texas A&M University. This 'decision, inherently 

related to regional and national accreditation, requi rements , ultimately 

sought to determine program effects and existing limitations in the various 

programs. More specifically,* the survey was designed to: 

1. determine the employment patterns and prpfessional profiles of 
former students from the graduate programs in Educational 

* Curriculum and Instruction and Educational Psychology. 

2. obtain perceptions of former students concerning the effectiveness 
of the graduate curricula in Educational Curriculum and Instruction 
and Educational Psychology. 

Survey methodology includes a variety of approaches (personal interview, 

telephone survey, mailouts) regardless of the function being served. Past 

experience with survey research has shown that while personal interviews 

yield the most cqmplete and valid ir#ormation\ the cost per data point ^ 

approaches the ratio- of 60/1 in comparison to mailout surveys (Smedley and ^ 

Olsen, 1975). Unfortunately, mailout surveys . of ten suffer from sampling 

bias because of low respons'e levels. However, Representative samples can 

be achieved if: ^ 

1. instrumentation is br-ief, clear afid concise. 

2. multiple mailouts are used and continuous updating of addresses 
is maintained. J ^ ^ ^ 

<v 3. phone contacts are made with those who 'have not responded after 
two mailouts (Rosser and Denton, 1976). 



ERLC 
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Given these considerations, a decision was made to pursue a mailout survey 
adhering to the forementioned recommendations. 
Subjects - ' 

Individuals receiving advanced degrees (M.Ed., M.S. and Ph.D.) from thfe 
departments oft Educational Curriculum and Instruction (EDCI) and ^Educational 
Psychology (EPSY) during the period from- May 1975 through May 1977 were , , 
selected as the sample* for this survey. This sample translates numerically 
to 279 individuals with departmental compositions being 122 and 157 for 
EDCI and EPSY, respectively. ■■ " 

Data Procurement ( , . • 

Names of individuals in this sample were identified from a complete 
listing of graduates on official graduation cfniiouncements . Lists, containing 
the name and address of each masters and doctoral recipient werte , developed 
from the "names listed in this source. 

The instrument developed for this survey was a&companied by a solicitation 
requesting the assistance^and ^cooperation from each graduate. The cover 
letter and questionnaire were mailed to the graduates on January 19, 1978. 
Five weeks were allotted for return of the completed instrument from the 
initial mailing. A second mail-out to non-respondents was initiated 
February 27 , 1-978 which contained a second cover letter over the signature 
of the Dean of the College of Education. A date six weeks hence was 
established as the final entry date for data received from the mailouts. 
After four weeks, phone numbers of non-respondents were obtained and phone 
contact was attempted' during March 28-29 from 5:30 p.m. to 8:00 p.m. 
individuals who were contacted in this manner were encouraged to return the 
questionnaire. On the closing date for receipt of questionnaires * 



(April 7, 1978), ?$me 145 completed questionnaires had been returned. 
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Nearly fifty-two percent of the advanced decree recipients who were 
identified in the sample responded on the - questionnaire. Table 1 summarizes 
►the returns by departmental affiliation and degree earned. 



Place Table 1 *about here 



Whil£ tj^e overall percentage of returns failed to # reach sixty percent, 
it is interesting to note that responses from doctoral recipients exceeded 
seventy percent for both departments. This discrepancy could be a result 
<_ of a stronger attachment to the college and its programs on the part of 

doctoral students. t x 

( ' ' / ■ ' < ' ' * 

Tnstrumentatzon* ' 

One instrument containing multiple components was developed to- colject 



perceptional data from the sample of graduate degree holders. The instrument 

; . w. 

was printed on card stock with the return address and postage label included. 

The initial component requested background information on the graduate 
regarding current professional roles, membership Mh professional organizations, 
professional publications, presentations', and recognitions. Component two, 
General Program Components , consisted of thirty-Jtwo items referenced to 
graduate program services and expected professional skills. . Each of these 
items .was referenced to 'two Likert-type scales. One scale requested a 
rating of the . necessity of . service on skill i^ the graduate program, while 
yie second scale sought a judgment concerning the qua! ity of service or 
instruction provided at Texas A&M regardi ng ^each of the thirty-two items. 
Alpha coeff iciettts of internal consistency were determined for each of the 
scales; i.e., necessity 'scale, .84, and quality scale, .89, both were statistically 
significant (p<.05). Examples of these scales and representative items are 
.presented in figure 1. • • . ' 



Place figure 1 about here 
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VThe 5 final Component,' Comments* consisted of open-response items addressing" t 
■ ' . % 

the strengths, weaknesses and needed. changes of the graduate programs administered 
though the -Col lege of Education. 

» *• 

Data Analysis 

Jfhe statistical procedures used to address the goals ' of the survey 
varied substantially., Efforts to determine empl oyment 'patterns and the 
professional accomplishments of graduates associated with goal number one 
were determined in a straight forward descriptive manner. Hcyever, the 

'mo^e involved^ This technique involved * 



analysis used for goal two wa< 
deriving a matrix score from the dual scale responses <of each graduate 
to each item. The^e matrix scores were subsequently classified in one of 
three categories based on the perceived difference between the importance 
and effectiveness scale ratings (Denton, 1978). 

The matrix score technique gets its name from organizing the responses 

A 

for an item into a 5x5 matrix, then continues by combining the matrix 
values into a total score. To illustrate, consider the response of one 
graduate to an item on the survey instrument. This individual marked the 
item very necessary (numerical value = 5) on the necessity scale and 
ineffective (numerical value = 2) on the effectiveness scale." The resulting 
tally in the matrix then appeared in the 2x5 cell. This procedure was 
repeated for each graduate resulting in a matrix with numerous tallies in - 
the various cells. A matrix score for the item then was determined by* 
converting the cell frequencies into percent values, multiplying the result-, 
ing cell percentage by the cell's decision-weight and summing the resulting 
values across all cells. These steps are' summarized by the 'fol 1 owing 
mathematical expression: 

Matrix scbre = ££f%(Rv - Cv) ' 



u 
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The f% represents the cell* frequency expressed as a percVntage and 
(Rv - Cv) represents the decision-weight for the cell. The numerical values 
of the row and column for. a cell is represented by Rv and Cv, respectively. 

This technique is illustrated in figure 2 with a hypothetical set of data 



Place Figure 2 about here " .■ 

2 — \ 

While the magnitude and sign of the matrix score depends on the 
decision weight used and the distribution of responses, cpmputatioTTTTT 
these values does not provide a means to rank the various., distributions . 
Since the foregoing procedure was applied to all items appearing on the 
dual .scaled instrument, an array of matrix scores resulted. A standard * 
based on the number of standard deviatiorf units from an optimum matifjx 
score of zero was used for ranking the responses across* items. The standard ^ 
devotion unit was determined by calculating the mean and standard- 
deviatioh of all matrix scores across four .graduate programs surveyed 



within the college. The number of matrix scores used in this computation 

was 128. Moreover, zero was selected as the .optimupi score because this value 
* * ♦ 

occurs when the ideal ratings* oh-'eacH-:. seal e (5) are substituted int£ the 
decision weight formula (Rv - Cv), i.e., 5 t 5 = 0. 

Since the questionnaire yielded information to assess the program 
components an arbitrary standard to rate the matrix scores was established 
consisting of three categories, namely, acceptable range (between - 1 S.D.). 
These categories translate into matrix scores of acceptable range (-44, to +44), 
review range (+/ - 45 to +/ - 88) and revise ^range (beyond - 88). Because 
these standards were established rather arbitrarily, broad bands were 
established to categorize the matrix scores. 

- • Y • 
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■ Find ings- - ^ 

' 'Due to the nature of this investigation, results- -of the various > 
analyses will be organized with respect to the goal s addressed by the 
follow-up evaluation. ^ 

Goal* I: ^mployment Patterns and Professional Profile of Graduates : J 

The professional roles of the respondents are presented in table 2. 
The majority of graduate degree recipients responding to, this questionnaire 

are engaged in professional education ,. that is, from 52% (EPSY-JD) to 

V ' * 

94,4% (EDCI-D); TheS-e employment profiles of Ph.D.. recipients vary * ■ 

considerably due to the number of graduates from EPSY who become counseling 

psychologists in agency "sibtti ngs rather than'accepting a position in V 

V< • - 

professional education; 1 < 



Place Table 2 'about here 
— % • — 

Other data requested of former sTudents were the nifmber of memberships 
in professional associations, leadership roles^.in those organizations , and 
honors bestowed by various groups. Further, information was sought on the 
number of publications achieved and presentations conducted~si nee completing 
the graduate degree. These data are summarized in Table 3. As expected, 
doctoral recipients across both departments were moref active, given these 
criteria, than their Q^nterpajjs with masters degrees. While the 
professional activities and productivity show promise" for these individuals, 
the reader is reminded that respondents to this survey has completed their 
degrl&s within 2 1/2 years. of the Survey: Thus, the average number of 
publications anefpresentations ar** relative values indicating the emphasis 
these individuals, and to'some degree their respective departments have 
placed on these measures of professional productivity. 
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* . ) , Place Tab l e" TTbout here 

Goal 2: Perceptions "Regarding Program Effectiveness 
Responses from recent graduates varied substantially when the data were 
grouped by preparation program and converted, to matrix scores^. Table 4 
.provides- the matrix scores for the 32 components assessed in this survey 
by departmental program^. ' 

Place Table 4 about here ' t y 

Surveying the matrix' scores reveals that' both positive and negative 
values resulted d^ue/to the weighting system, (Rv - Cv), used to' 
compute the scores.- Negative ,scores occurred when the necessity of the service/ 
skill was perceived to be of greater importance than the effectiveness of 
the "program to provide quality instruction for the service/sjjl 1 . Conversely, 



positive matrix scores resulted when the effectiveness of the^pr^ram in 
offering the secvice/skiTl was perceived to be greater than thr necessity 
for that particular service/skill. 

Summarizing the findings we see from table 4 the masters degree 
respondents from both EDCI and EPSY were more critical of their programs than 
their counterparts with* doctorates . In all casgs where program limitations 
were found to occur, graduates perceived thequal.ity of the curriculum. 
associated*with certain services/skills to be limited given the relative 
importance of the skill. Some 11 components of the curriculum were perceived 

' to be fn^need of revision by the graduates from EDCI. These components were: 

Program advisement. with degree planning 
Support for Participating in Professional Conferences 
Job Placement 
Problem Solving 
Teaching 
• , Program Evaluation 

Professional Writing \ 
Administration 

Supervision * 
% Human Relations with Colleagues^ 
•Professional Speaking 



ERIC 
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v Written comments from graduates at the masters iTeveJ tended ^either toi 

laud some«courses'»i .e. , microteaching, reading coursework and curriculum 

development or requested additional course work, e.g., statistics, confent 1 ^ 

field methods courses. In addition'', a number of masters' degree respondents 

stressed "the need for greater emphasis on application types of experiences: 

The doctoral .respondents lauded Individuals and the department ih general. 

for good teaching. Conversely, a number of individuals suggested more* 

background in applied statistics, computer, science and research design as 

PQSsibil i ties that should be considered. One additional skill, ^how to 

publish" was mentioned more* than once by the doctoral recipients. 

Analyses of the responses from EPSY graduates yielded somev8 curricular 

m 

components perceived to be in need of revision. The components were:' 

* * Library Holdings i 

Job Placement 

Support for Participating in Professional Conference 
Problem Solving J 
ProgranrEval uation - / 

Administration * 

Human Relations with Colleagues ; ... 

_ Professional Speaking 

In general, responses from, decent graduates to the open-ended items 

on the questionnaire support thfe preceding list of program components in ne^d 

of revision. Both masters and doctoral recipients remarked that additional 

practi^a sh^il<i be included in the EPSY programs; and that library services . 

were substandard . 'In addition, a number of masters level respondents voiced 

displeasure with the advice they received in degree planning the value or th^ 

final examinations. Conversely a number of remarks commended individuals and 

the department for f i ne^teachi ng , especially in the areas of * counsel ing , 

'statistical applications and research design. In general, the only recurring 
* 

concern at* the doctoral Jevel centered on the need for more emphasis in the 
psychological foundations. 
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' f ' ' I. 

Discussion • n 

if 



Both the status 'function and relational functjoijbf surveys attributed 
to Kerlinger's (1964) discussion o,f surveys were addjflssed in this' project. 

-• ~ . if; 

The descriptive information on the professional positions and productivity 

» ... .|f ■ ' . . 

of graduates 'aptly fits the status Vunction , while fhe matrix scores^*® 

resulting 'ratings of program components address th| relational function. 

% ;; \F ■ 

The descriptive information krthis survey preVided^ a contextual basis 

< } 
for assessing the perceptions of former students regarding 'the quality of 

'program services and skills. To illustrate, consider the fact that both 

EDCI and EPSY graduates perceived services associated with job placement 

.to be wanting; yet with the possible exception^ of "EPSY doctoral recipients, 

a convincing percentage of graduates from both departments were employed in 

^ ■ 5 0 ■ 



positions^ professional education. > Thus, this concern N^garding job 

placement takes on a different meaning than^Jf the majority of graduates 

had been unemployed or employed in non-educational positions. Another 

example' to illustrate the value of descriptive data relates to problem 

solving skills. Graduates from both departments indicated they were not 

adequately prepared for problem solving'tas^s'iiven their responses on the 

two scale item on 'the instrument, Subsequent reviews of comments from the 

instrument and a review of the various curricula have revealed .this cognitive 
-*'* 4 '-'ft 

■skill is not adequately addressed across.the various programs. 

> The findings related to the, matrix scores of this investigation reveal 

substantial differences do exist, at least jj the perceptions of recent 

graduates, regarding the vaVious graduate programs .experienced. Yet the 

degree to whiph-tbese perceptions reflect actual deficiencies in preparation 

with respect- to the professional ehvironmen|< they are working in is unknown. 

Thus,* the -perceived relational, deficiencies associated with the matrix scores 

nee,d to be examined with- respec|f o the ;pj|of esjiqiial characteristics and: 
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and productivity of the graduates- and other measures of the respective 
curricula before decisions can be rendered about the quality of the 
curricula. 



\ 
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necessity of this 
skill is: 



CO 

n 



1 



1 



Skills In; 
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Program Evaluation 



EFFECTIVENESS 

My preparation on this 
skill was: 
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Figure 1 

Illustrative .Items from the Two-Scaled 
Survey "Instrument 
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F%(Rv-Cv) 

10(1-5) = 

10(2-2) = 

20(2-4) = 

10(2-5) = 

20(3-3) = 

10(4-4) = 

20(4-5) = 



-40 
0 

-40 
-30 
0 
0 

-20, 



Matrix ZZ^Rv-Cv) = -130 
Score 



Figure 2 

Calculation of Matrix We from Hypothetical Set of 10 
^ ' Responses on^ Two Scaled Instrument 




Table 1 



Summary of Return From Follow^Up Survey 
of Graduate Programs in EDCI and EPSY 



Response by - Number Number Percent Returned 

Department Sent Returned' Compl eted 



Ed. Curriculum 4 Instruction 


122 


63 




51 


6 


Masters 


95 


43 




X 45 


3 


Doctoral 


27 


20 




v 74 


1 " 


Educational Psychology 


157 


82' 




52. 


2 


Masters 


124' 


58 




46 


8 














Doctoral 


33 . 


24 




72 


7 * 




4 











Table 2 



Current Professional Positions of Graduate Degree Recipients 
Expressed as Percentages in Eagh^Department 

) 



Ed . Curricul urn 
& Instruction 



' IT 

Educational 

Psychology 



Position 



M 



D 



M 



D 



Teacher (P.S. ) 

Teacher (H.E.) 

Counselor (P.S. ] 

Counselor (H.E. [ 

Consul tant 

Admi nistrator 
•(P.S.J 

Administrator 
(H.E.) 

Supervisor 

Evaluation 
Speacial 1st 

Researcher 

Other 



65 

ll.fr 



4.7 



11.1 
44.4 



11.1 
11.1 

5.6 



2.3 *6.6 

5.6 

4.7 

11.6 $.6^** 



44 
3.5 
24.6 

3.5 
1 .8 

3.5 



21.7 

17.4 
4.4 

4.4 



3.5 

1.8 4.4 



14.0 47.8 



M = Masters degree recipient 
D = Doctoral degree recipient 



P.S. = Public School 
H.E. 2 Higher Education 




Table 3 



Summary of Professional Activities 
Average Number of: Organizational Memberships, Organisational Offices, 
Publications, Presentations, and Honors Bestowed on Advanced Degree Recipients 



Advanced Degree Recipients 



Variable/graduate 



Average # of r Membershi ps 
Average f of Offices 

% 

Average $ of Publications 
Average 0 of Presentations 
Average # of Honors 



M s 'Masters degree recipient 
D = Doctoral degree recipient 



Ed, 


Curr. & 


Educational 


Instruction 


u Psychology 


M 


D 


j " 


D 


2.40 


4.40 


2.40 > 


3.30 


.40 


1.20 




1.10 


.26 


1.70 




2.30 


.61 


2.70 


.44 


1.60 


.28 


.17 


* .12 


.70 
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Table 4 



Matrix Scores for Program Services and Professional SkUls-by Department an/ Degree 



Service/ Skill 



Program advTs^ment with degree planning 
Proaram ArlvicUmnnf uiffh .•*.«. 



Program advisement with advisory committee 
counsel 

masters thesis and 



masters oral exam 
doctoral prelims 
dissertation defense 



\ 



•Final examination: 
oral 

Final examination: 
Final examination: 
Final examination: ~ . u , , WP , 
Helpfulness of principal advisor 
Helpfulness of departmental faculty * 
Library ^holdings 
Job placement 

Financial support 

Support for participating in professional 

conf. 
Computer services 
Research laboratories 
Research equipment 
Access to original sources of data 
Problem s^oWing 
Teaching * 
Program evaluation 
Professional writing 

Administration ^ 
Supervision 

Research: experimental design 
Research: literature searches 
Research: statistical applications 
Research: sampling 
Research: data presentation 
Research: documentation of findings 
Human relations with students and clients 
Human relations with colleagues 
Professional speaking 
Scholarship 



M = masters degree 
D = doctoral degree 
( ) low number of responses 
* denotes scores occurring in review category 
denotes scores occurring in revise category 



Ed. Curr. & 
Instruction 



-100** ^65*~ 
- 69* * -41 

(100) (100) 
15 (250) 



-63* 
-63* 
-80* 
-91* 



25 
18 

-17 

-33 

-94* 

-60* 



Educational 
_ Psychology 

M D 



-75* 
-48* 

(107) 
28 

(120) 

(100) 
-40 
-50* 

-102** 
-89** 



13 
9 

(50) 
( 0) 
-13 
35 
-17 

-u 

-126** 
-125** 



-32 


-44 


-65* 


-50* 


-44 


-143** 


-54* 


-109** 


16 


-33 


(-75) 


-18 


-53* > 


25 


(-6) 


-72*. 


-18 


(-14) 


(-23) 


-83* 


-21 


-46* 


-67 


-88* 


-115** 


-76* 


-107** 


-59* 


-90** 


-38 


-71* 


-32 


-100** 


-44 


-110** 


-68* 


-109** 


-35 


-55* ^ 


' -67* 


-122** 


6 


-78* 


-111** 


-123 


-6 


-82* 


-50* ' 


-44 


-41 


5 


0 


0 


-76* 
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